Papers with systematic generalization
Systematicity Emerges in Transformers when Abstract Grammatical Roles Guide Attention (2022.naacl-srw)
Copied to clipboard
| Challenge: | Existing systems that use transformers lack systematicity, but they are inferior to human learners in sample efficiency and difficult generalization problems. |
| Approach: | They propose to modify a transformer so that it controls attention distributions and fills in the gaps. |
| Outcome: | The proposed model shows that the performance of natural language processing systems is improved when abstract role labels are assigned to the input stream and provided to the role stream. |
Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering (2021.tacl-1)
Copied to clipboard
| Challenge: | Neural networks fail to generalize to out-of-distribution examples that contain new compositions. |
| Approach: | They propose a model that computes a representation and denotation for all question spans in a bottom-up, compositional manner using a CKY-style parser. |
| Outcome: | The proposed model achieves 96.1% accuracy on a challenging dataset compared to baseline models on . previous models failed to generalize to out-of-distribution examples . |
Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inference (2022.tacl-1)
Copied to clipboard
| Challenge: | a neural network model for natural language inference (NLI) is proposed. |
| Approach: | They propose a neuro-symbolic natural logic framework based on reinforcement learning with introspective revision that rewards specific reasoning paths through policy gradients. |
| Outcome: | The proposed model shows superior capability in monotonicity inference, generalization, and interpretability compared with previous models on the existing datasets. |
Compositional Generalization by Factorizing Alignment and Translation (2020.acl-srw)
Copied to clipboard
| Challenge: | a crucial property underlying the expressive power of human language is its systematicity. |
| Approach: | They propose to make an analogous separation between alignment and translation in neural machine translation to capture compositional structure. |
| Outcome: | The proposed architecture outperforms existing neural networks on a compositional generalization task without supervision. |
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that basic configurations can improve the performance of neural networks on systematic generalization. |
| Approach: | They propose to revisit basic configurations to improve the performance of Transformers on systematic generalization by revisiting scaling of embeddings, early stopping, relative positional embeddment, and Universal Transformer variants. |
| Outcome: | The proposed models improve accuracy from 50% to 85% on the PCFG productivity split and from 35% to 81% on COGS. |
Systematic Generalization on gSCAN with Language Conditioned Embedding (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing deep learning models fail when the test set is systematically different from the training data. |
| Approach: | They propose a method that explicitly models the relations between objects in their contexts while learning their representations. |
| Outcome: | The proposed model outperforms the baseline model and reaches state-of-the-art performance on grounded SCAN (gSCAN), a grounded natural language navigation dataset. |
Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have highlighted severe limitations of contrastive learning models in their ability to perform compositional reasoning over objects, attributes, and relations. |
| Approach: | They propose a graph decomposition framework and negative mining techniques to improve attribute binding and relation understanding of scene graphs. |
| Outcome: | The proposed approach improves attribute binding, relation understanding, generalization, and productivity on multiple benchmarks. |
Systematic Generalization in Language Models Scales with Information Entropy (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for assessing compositional behavior are unclear on how to measure the difficulty of a systematic generalization problem. |
| Approach: | They propose a framework for measuring entropy in a sequence-to-sequence task and a method for measuring it. |
| Outcome: | The proposed framework scales with the entropy of the distribution of component parts in the training data. |
Learning from Task Descriptions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in supervised NLP has shown significant progress in learning tasks from examples. |
| Approach: | They propose a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. |
| Outcome: | The proposed model achieves 12% on the new dataset, leaving a significant challenge for NLP researchers. |
Challenges in Generalization in Open Domain Question Answering (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work on Open Domain Question Answering has shown that there is a large discrepancy in model performance between novel test questions and those that largely overlap with training questions. |
| Approach: | They introduce and annotate questions according to three categories that measure training set overlap, compositional generalization, and novel-entity generalization. |
| Outcome: | The proposed models perform better on established datasets and lower on comp-gen/novel-entity questions than on the full test set. |
Systematic Generalization on gSCAN: What is Nearly Solved and What is Next? (2021.emnlp-main)
Copied to clipboard
| Challenge: | a general-purpose Transformer-based model with crossmodal attention solves most of the systematic generalization problems . current models are data inefficient given the narrow scope of commands in gSCAN . |
| Approach: | They propose to use a Transformer-based model with cross-modal attention to solve gSCAN . they propose to generate data to incorporate relations between objects in the visual environment . |
| Outcome: | The proposed model outperforms specialized approaches on most splits, and is data inefficient given the narrow scope of commands. |
Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategies (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized the field of natural language processing . however, it has been shown that they lack systematic generalization, which allows to extrapolate the learned statistical regularities outside the training distribution. |
| Approach: | They propose to benchmark a LLM with two parameters to find out its performance . they compare it to a variant of the Transformer-Encoder architecture to find the same problem . |
| Outcome: | The proposed model outperforms the previous model on three algorithmic tasks with two parameters. |
LAGr: Label Aligned Graphs for Better Systematic Generalization in Semantic Parsing (2022.acl-long)
Copied to clipboard
| Challenge: | Semantic parsers struggle to generalize to examples with unseen combinations of seen rules from the training set. |
| Approach: | They propose a general framework to produce semantic parses by predicting node labels for a complete multi-layer input-aligned graph. |
| Outcome: | The proposed framework produces better generalizations than the baseline framework . it produces representations directly as a graph and not as sequences . |
SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation (2024.acl-long)
Copied to clipboard
| Challenge: | Popular neural architectures lack strong structural inductive biases for seq2seq NLP tasks . previous work shows that these models struggle with systematic generalization . |
| Approach: | They propose to inject a structural inductive bias into a seq2seq model by pre-training it to simulate structural transformations on synthetic data. |
| Outcome: | The proposed method improves few-shot learning and generalization of FST-like models. |
Lexicon Learning for Few Shot Sequence Modeling (2021.acl-long)
Copied to clipboard
| Challenge: | Past work has shown that many failures of systematic generalization arise from neural models’ inability to disentangle lexical phenomena from syntactic ones. |
| Approach: | They propose a lexical translation mechanism that generalizes existing copy mechanisms to incorporate learned, decontextualized, token-level translation rules. |
| Outcome: | The proposed model improves generalization on a diverse set of sequence modeling tasks drawn from cognitive science, formal semantics, and machine translation. |
CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text (D19-1)
Copied to clipboard
| Challenge: | Existing datasets for reading comprehension tasks have been used to test the generalization of natural language understanding systems. |
| Approach: | They propose a diagnostic benchmark suite to clarify key issues related to the robustness and systematicity of NLU systems. |
| Outcome: | The proposed benchmark suite clarifies key issues related to the robustness and systematicity of NLU systems. |
Balaur: Language Model Pretraining with Lexical Semantic Relations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks. |
| Approach: | They propose a model that models LSRs directly in the LM’s hidden states throughout pretraining by allowing them to be controlled by the model. |
| Outcome: | The proposed model improves performance on hypernymy-informed tasks and on the original LM objective. |
Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Survey (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks and models focus on systematicity of representations, but they focus on the systematicity in behaviour. |
| Approach: | They argue that systematicity is a desirable property in ML models as it enables strong generalization to novel contexts. |
| Outcome: | The proposed benchmarks and models focus on the systematicity of behaviour, while existing models focus primarily on language and vision. |